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ABSTRACT 

The purpose of this study was to explore the relationship of problem-based 
learning (PBL) and the development of critical thinking disposition (CT) and 
academic achievement in Chinese medical students using a cross-sectioned 
randomized design. Medical students from China Medical University (CMU) were 
randomized to PBL or non-PBL teaching at the commencement of the study. After 
five years of study, CT was scored by a Chinese version of the California Critical 
Thinking Disposition Inventory (CCTDI-CV). The score achieved on a Computer 
Case Simulation (CCS) test evaluated academic performance. Total CT score was 
higher in PBL students (n=170) than non-PBL students (n=83) (304.7+36.8 vs. 
279.2+39.4, p < 0.01). Subscale CT-scores were significant in favor of PBL in six 
of the seven subscales (truth seeking, open-mindedness, analyticity, systematicity, 
inquisitiveness, maturity). There was no significant difference in terms of gender 
on the total CT score, though minor differences were seen in subscales favoring 
female PBL students. PBL students had higher CCS scores than non-PBL 
students, but not significantly (112.8+20.6 vs. 107.3+16.5; p=0.11). There was no 
significant correlation between CCS scores and CCTDI-CV results. Male students 
scored slightly higher on the CCS test compared to female students (male 
113.4+18.9 vs. female 109.7+19.7), but the difference was not significant. This 
study concludes that in Chinese medical students, PBL teaching was related to a 
higher disposition of critical thinking, but not to improved academic skills. 
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INTRODUCTION 

Critical thinking (CT) is increasingly regarded as an essential element of educational activities 
and is defined as a prerequisite skill for health and medical related professions, in particular. 
CT skills have been designated as one of the desired major learning outcomes of medical 
schools - as both an academic capability and a professional competence (GMC, 2003; The 
Tuning Project Medicine, 2005). 

Definitions of CT vary and include aspects of psychology, cognition and philosophy. Watson 
and Glaser (1964) defined CT as a combination of attitude, knowledge and skills. Brookfield 
(1987) classified four elements of CT: 1) identifying and challenging the assumptions that 
serve as the basis of ideas, values and actions; 2) challenging the importance of context; 3) 
assuming and exploring the alternatives; and 4) reflective skepticism. According to Facione 
and Facione (1996), CT is a nonlinear and cyclical process that allows people to make 
decisions on what to believe and what to do within a given context. 

A quantitative assessment of critical thinking can be used as an instrument to guide education. 
The California Critical Thinking Disposition Inventory (CCTDI) (Facione and Facione, 1992) 
was developed to measure students’ dispositions towards CT. CCTDI took as its point of 
departure the Delphi report on CT by the American Philosophical Association Delphi Panel, 
which defines CT to be “purposeful, self-regulatory judgment, which results in interpretation, 
analysis, evaluation, and inference, as well as explanation of the evidential, conceptual, 
methodological, criteriological, or contextual considerations upon which judgment is based” 
(Facione, 1990). Since its development, CCTDI has been well employed to assess student CT 
disposition and skills in a variety of programs in health and medical education (Leaver-Dunn 
et al., 2002; Tiwari et al., 2006). 

Previous studies not only focused on the measurement and assessment of students’ CT, but 
also on its correlation with academic performance, students’ success and so on. Results of 
these studies have been used in program evaluation and curriculum improvements (Phillips 
and Rospond, 2004). 

Previous scholars have asserted the importance of the learning environment in relation to the 
development of CT in university students. Problem-based learning (PBL) has been used as a 
strategy for promoting CT because it addresses the ability to analyze and solve more complex, 
real-life problems (Oja, 2011). With its theoretical roots in constructivism, PBL has been well 
used as an educational strategy and method in medical education since late 1960s. By 
promoting efficient knowledge acquisition, self-directed learning, participation, critical 
thinking, self-reflection and evaluation, PBL methodology has driven innovative curriculum 
change and educational reform in many medical universities in the past more than 40 years. 
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An extensive literature has documented sufficient cognitive evidence to validate the PBL 
approach to learning outcomes as well as its effectiveness in improving students’ clinical 
performance (Duch et al., 2001; Neville, 2008). 

The PBL approach has been described as an effective and efficient strategy to encourage 
students to improve analytical, problem-solving and collaboration skills (Du et al., 2010), and 
therefore seems well suited to building critical thinking skills (Oja, 2011). A number of 
studies have shown a positive relationship between PBL and critical thinking in nursing 
education (IP et al., 2000; Tiwari et al., 2006; Yuan, Williams and Fan, 2008, Jones, 2008; 
Oztiirk et al., 2008). These studies showed that nursing students in the PBL group scored 
significantly higher on the CCTDI than students in the non-PBL group. 

Despite numerous studies on PBL effectiveness in various aspects of medical education, 
studies on the effect of PBL on CT skills of medical students remain limited, possibly because 
of the demanding and complex nature of medical education for both students and teachers. As 
a consequence, the majority of the CT literature consists of studies on students from the fields 
of nursing and other health-related study programs. In particular, comparison of PBL and 
other teaching and learning methods is sparse (Worrell and Profetto-McGrath, 2007). 
Previous studies have also questioned whether PBL is compatible with the Chinese culture of 
learning and teaching (Lee et al., 2004) and how Chinese learners may improve critical 
thinking by PBL (Tiwari et al., 2003). Therefore there is a need for more empirical studies to 
support the assumption that PBL improves CT in students in health sciences and, in particular, 
in medicine, in non-Westem contexts (Yuan et al., 2008; Oja, 2011). 

PBL as an educational strategy has been used in medical education in China since the late 
1990s; however, the majority of PBL educational practices have been implemented by 
individual teachers interested in PBL rather than at the institutional level. 

In 2004, China Medical University (CMU) started an educational reform project focused on a 
systematic implementation of PBL. In 2004, pilot experiments were carried out with a small 
number of student groups and in the autumn of 2005, PBL methodology was introduced into 
the majority of the medical study programs at CMU. PBL implementation at CMU has been 
practiced as a hybrid model (Savin-Baden, 2003) in which diverse teaching methods co-exist. 
Such teaching methods include clinical cases, lectures in traditional discipline courses, 
lectures in integrated courses, student group discussions, tutoring and web-based discussions. 
At the same time, non-PBL programs centered on traditional lecturing and exercise modalities 
were continued alongside PBL learning, providing a unique opportunity for conducting a 
randomized controlled trial on the effect of PBL on the learning and development of critical 
thinking in a complete medical program. Previously, randomized controlled trials have been 
used to study the effect of PBL and self-directed learning for specific topics such as evidence- 
based medicine, with the result for PBL being negative in terms of learning outcomes 
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(Johnston et al., 2009; Bradley et al., 2005). The adoption of PBL by a large Chinese 
university is interesting from a cultural perspective as it has been speculated that Asian 
culture does not facilitate self-directed and critical learning, core principles of PBL (Tweed 
and Lehman, 2002). 

The PBL implementation at CMU was dependent on continued interest and engagement in 
pedagogy reform from both teachers and students. By 2009, more than 1,000 staff and 10,000 
medical students had been involved in PBL, with student participation from years five, six, 
and seven of the programs. 

The present study was conducted from 2005 to 2009 to investigate the effect of PBL on 
CCTDI and academic performance as evaluated by a computer case simulation (CCS) test 
from National Board of Medical Examiners (NBME). 

METHODS 


Research questions 

The study design was constructed to address the following research questions: 

• What is the CT disposition in medical students in a PBL environment compared with 
those in a non-PBL environment? 

• Are there differences between female and male students with respect to the development 
of CT? 

• Is there a difference in CCS grade points between PBL and non-PBL students? 

• Is there an association between academic achievement as measured by CCS grade point 
and CT in PBL and/or non-PBL medical students? 

Tools, CT 

To estimate CT, CCTDI, a 75-item Likert scale tool, was used to measure students’ CT 
disposition (Lacione and Lacione, 1992). CCTDI has seven scales: truth seeking, open- 
mindedness, analyticity, systematicity, CT self-confidence, inquisitiveness and cognitive 
maturity. Total scores range from 70 to 420, with marks above 280 indicating a positive 
overall CT. In each disposition scale, a score of 30 or below indicates a consistent negative 
disposition or weakness in relation to the given attribute or characteristic and a score of 40 
indicates a positive disposition for the attribute on average, and a total sum above 280 
indicates a positive disposition. 

Scholars from Hong Kong Polytechnic University (Peng et al., 2004) developed a Chinese 
version of the CCTDI (CTDI-CV). Rather than focusing on word-for-word equivalence in the 
Chinese translation, CTDI-CV is focused on conceptual equivalence to CCTDI (Peng et al., 
2004). In consideration of cultural sensitivity, modifications were made. CTDI-CV has been 
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used as an instrument to study CT in a growing number of universities in China. Due to 
timing and consideration of resources, the CTDI-CV could not be used at the start of the 
present study. Therefore, a baseline for CT development is not available and comparisons 
were only made cross-sectionally. 

Tools, Academic Achievement 

In 1999, CCS was introduced into Step 3 of the United States Medicine License Examination 
(USMLE). A Chinese version of CCS was introduced at CMU in 2002 as a major graduate 
assessment method for five-year and seven-year track students at the end of their fifth year of 
study. By 2010, 4,000 students had participated in this test. In CMU practice, this test 
included two stations with random clinical cases. At the time of testing the database contained 
80 cases in Chinese, with a time limit set to 20 minutes for each case. 

Participants 

All students were aged between 18 and 22 years at admission and had high admission grades. 
The study was conducted among students from the clinical medicine program. Students were 
enrolled in the seven-year program for a master’s degree in medicine in September 2004 
(n=270). All students achieved their university placement based on the scores from their 
National College Examinations. They were evenly and randomly divided into nine classes 
containing thirty students each. The nine classes were all comparable in terms of students’ 
ages, gender ratio and academic scores in 2004. Beginning in September 2005 (the second 
academic year), PBL methodology was implemented in six out of the nine classes. The 
remaining three classes had no PBL implementation. 

After 5 years of study, 267 students took the CCS test between September 10 and 20, 2009, 
and a CCTDI test was conducted on September 30, 2009. The results of these tests were used 
as the data sources of this study. Out of 267 questionnaires distributed, 256 were returned 
(female 169 and male 87). Among these, 170 students (female 110 and male 60) from 6 
classes had PBL-based courses (PBL students) and 83 students (female 58 and male 25) from 
3 classes had no PBL experiences (non-PBL students). 

Statistics 

Data were analyzed using SPSS 19.0. MANOVA was used to analyze the effects of teaching 
method and gender on critical thinking scores and CCS scores, with teaching method (PBL 
vs. non-PBL) and gender as independent variables and CT and CCS scores as dependent 
variables. A p-value of less than 0.05 was considered significant using Wilks’ Lambda in the 
multivariate analysis. Model assumptions were tested by analyzing data for outliers and using 
Box’s test for equality of covariance and Levene’s test for homogeneity of variances. 
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RESULTS 

Students were comparable in age, gender and admission scores. There was a positive effect of 
PBL on CT scores, as PBL groups showed a total CT score of 304.7 and non-PBL groups a 
score of 280.4 (see Table 1). Female students scored 307.6 (PBL) vs. 281.6 (non-PBL) and 
male PBL students scored 299.4 vs. 277.6 for non-PBL students (see Table 1). 

The overall multivariate analysis of variance showed a significant effect for learning method 
(PBL vs. non-PBL) (p=0.00, F=13.02), but not for gender (p=0.19, F = 1.69). 

Analysis of variance for each group (tests between subjects) showed the overall model to be 
significant (p=0.00, F= 3901.96). For the learning method, total CT score was significantly 
different (p= 0.01, F=23.76) but the CCS score was not (p=0.11, F=2.57). For gender, neither 
the total CT score nor the total CCS score was significant (p=0.32 and p=0.12 respectively). 

Individual components of the critical thinking tests were also assessed by multivariate 
analysis, including all seven subgroups of critical thinking as dependent variables and 
learning method and gender as independent variables. The multivariate test showed that the 
learning method produced a significant difference (p<0.005, F=4.54) but gender did not 
(p=0.34, F=1.14). 

For the learning method, analysis of variance for each critical thinking group (between- 
subjects test) showed all sub-items except self-confidence (p=0.173, F=1.86) to be significant 
for all students. 

Table 1. Descriptive statistics, showing means and standard deviation of CT disposition 
scores as a function of teaching method, females and males. For every item in the critical 
thinking inventory, there is an increased score in the PBL group versus the non-PBL group. * 
indicates significant difference (p<0.05). 



All students 

Female 

Male 

Learning Method 

Non-PBL 

PBL 

Non-PBL 

PBL 

Non-PBL 

PBL 

N 

83 

170 

58 

110 

25 

60 

Truth seeking 

38.6+7.1 

41.4+6.4* 

39.2+7.0 

41.5+6.1 

37.2+7.2 

41.2+6.8 

Open-Mindedness 

39.8+7.5 

44.1+6.3* 

40.0+7.6 

44.7+6.3 

39.5+7.2 

42.9+6.3 

Analyticity 

41.7+8.1 

45.2+6.8* 

42.3+7.4 

45.8+6.1 

40.3+9.5 

43.9+7.8 

Systematicity 

38.4+6.8 

41.5+7.0* 

38.9+6.7 

41.0+6.5 

37.2+7.0 

42.4+7.8 

Self-confidence 

40.7+9.1 

42.0+7.9 

41.0+9.0 

41.4+8.0 

40.2+9.7 

43.1+7.7 

Inquisitiveness 

42.0+9.1 

46.5+8.2* 

42.2+9.1 

46.9+8.2 

41.4+9.6 

45.7+8.4 

Maturity 

38.0+8.8 

44.2+7.6* 

37.8+8.8 

44.7+6.9 

38.1+9.1 

43.3+8.6 

Total 

279.2+39.4 

304.7+36.8 

281.4+40.1 

305.9+34.3 

274.6+43.3 

302.5+41.3 
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DISCUSSION 

Results from this study identify a general positive effect of the PBL method on CT disposition 
in Chinese medical students after their fifth year bachelor program study at CMU. This study 
observed findings similar to studies on pharmacy students (Phillips et al., 2004) and on 
nursing students (IP et al., 2000; Tiwari et al., 2006; Oztiirk et al., 2008; Yuan et al., 2008; 
Yuan, Williams and Fan, 2008). 

Findings of the present study are consistent with previous research results that PBL has a 
positive impact on students’ learning in terms of CT disposition and its improvement in 
different social, cultural and discipline contexts (Yuan et al., 2008; Oja, 2011). Positive CT 
disposition results of PBL-students can be attributed to the characteristics of this method. 
Using problems as a starting point for learning can potentially increase curiosity and 
eagerness to acquire knowledge and explore reasons and explanations. When starting from 
problems, students must develop self-directed learning strategies, which may help to improve 
their ability to relate theory to practice, analyze, evaluate, judge, conclude and make decisions 
(Hmelo and Lin, 2000). Group discussion and teamwork provide good opportunities to be 
open to a variety of opinions, and thus develop an increased tolerance for other’s viewpoints. 
In the process of managing self-directed learning through teamwork, students improve their 
maturity (Schmidt and Moust, 2000). 

As reported in the results, the PBL group scored positive (above 40) on all the seven 
subscales of CCTDI. The non-PBL group scored positive (above 40) on four of the seven 
subscales: open-mindedness, analyticity, self-confidence, and inquisitiveness, and scored 
negative (below 40) on three subscales: truth seeking, systematicity and maturity. The PBL 
group scored significantly higher than the non-PBL group, which may be an effect of 
employing the PBL methodology. 

The PBL group demonstrated significantly higher scores than non-PBL students in six out of 
the seven subscales: truth seeking, open-mindedness, analyticity, systematicity, 
inquisitiveness and maturity. This is consistent with other results involving Chinese students 
(Ip et al., 2000; Marcia et al., 2003; Yuan et al., 2008; Tiwari et al., 2006). 

Lower scores in truth seeking and systematicity were also found in previous studies of nursing 
students studying in lecture-centered environments in the U.S., Australia and Japan 
(Kawashima and Petrini, 2004; Marcia et al., 2003). This is consistent with the summary of 
PBL effectiveness on critical thinking in comparison with lecture-based study environments 
(Oztiirk, 2008; Oja, 2011). 

This study also mirrors previous studies (Ip et al., 2000; Marcia et al., 2003; Yuan et al., 
2008; Tiwari et al., 2006) concerning the subscale self-confidence: the score did not differ 


78 



XiangYun Du et cil 


JPBLHE: VOL. 1, No. 1, 2013 


significantly between the PBL and non-PBL groups, although both groups scored positive 
(above 40). To a certain degree, this may indicate that there are other factors that have not 
been included in this study influencing students’ self-confidence or that self-confidence is not 
as closely related to CT for this study group as the other subscales in the inventory are. 

A previous study documented a correlation between CCTDI scores and academic 
performance (Yuan, Williams and Fan, 2008). However, the correlation between CCTDI and 
CCS results was not significant in this study. It is interesting to note that PBL did not have a 
positive effect on academic performance, as this corroborates emphasizes previous skepticism 
of using PBL in China (Lee et al., 2004). However, further investigation is needed to examine 
the correlation between CT and academic performance. 

Results of this study identified no statistically significant gender differences either in relation 
to CCTDI or to academic performance based on CCS scores, except for a small advantage for 
male students in the CCS test. This finding echoes previous studies in which no overall 
gender difference could be found in relation to CCTDI results (Yuan, Williams and Fan, 
2008). 


CONCLUSION AND PERSPECTIVES 

In summary, the PBL environment as organized at the Chinese Medical University had a 
positive impact on CT disposition and may improve performance of Chinese medical 
students. This study suggests that medical education in general is challenged by new 
demands. Changes in the demands of the medical profession make critical thinking an 
increasingly important competence. The random allocating of students to classes in which 
teaching methods are different can potentially impact students’ learning; nevertheless, 
evidence-based approaches to educational experiments and innovation are important. The 
implementation of educational innovation is done in a multi-professional environment where 
relational skills and organizational understanding are crucial. Furthermore, longitudinal 
follow-up studies on the medical careers of students from both groups of students will later 
show whether the different learning environments and increased disposition towards critical 
thinking has an effect on postgraduate performance. Follow-up investigations on patient 
treatment outcomes for each group of future medical doctors could complement this study. 
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